Image pickup apparatus, and control method thereof

ABSTRACT

An image processing apparatus obtains an image signal of an image pickup element having a pupil division unit that restricts light of an optical image of an object arriving at each pixel of the element to light from a specific pupil area of a photographing lens. A memory unit stores information of a pixel defect of the element. An image shift unit determines a shift amount of the image signal of the photographed object corresponding to an image re-formation plane for every pupil area, to shift the image signal on the basis of the plane. A defect correction unit corrects an image signal of a defective pixel using the shifted image signal of a pixel other than the defective pixel, in accordance with the information of the pixel defect. An image re-formation unit re-forms an image corresponding to the image re-formation plane from the corrected image signal.

CLAIM OF PRIORITY

This application is a divisional application of copending U.S. patentapplication Ser. No. 13/705,992, filed Dec. 5, 2012, which published asU.S. Patent Application Publication No. 2013/0162866 A1 on Jun. 27,2013.

This application also claims the benefit of Japanese Patent ApplicationNo. 2011-277119, filed on Dec. 19, 2011, which is hereby incorporated byreference herein in its entirety.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image pickup apparatus having animage pickup element, such as a digital camera, and more specifically,to an image pickup apparatus having a function of correction processingof a pixel defect of an image pickup element.

2. Description of the Related Art

In an image pickup element that is used in a digital camera, it is verydifficult to prevent a pixel defect itself from being formed. In themeantime, there is an increasing need for measures against the pixeldefect due to a tendency toward a larger number of pixels (an increasein the number of defects) and a tendency to more narrow a pixel pitch(an increase in a defect ratio), in recent years. As characteristicsrequired for the processing of the pixel defect, a high capability ofprocessing a pixel defect and an appropriate processing load may begiven, for example.

Further, an apparatus (called a light field camera), that re-forms animage by obtaining lights passing through different pupil areas and usesthe re-formed image as an output image, has been suggested.

For example, Japanese Patent Application Laid-Open No. 2007-4471discloses a method of picking up an object image by using an imagepickup element that is capable of individually receiving lights passingthrough different pupil areas, and creating an image whose focus isadjusted after image pickup.

Further, “CCD camera technology” written by Yasuo Takemura, RadioTechnology Corporation discloses a method of interpolating the pixeldefect (details of the pixel defect will be described below) inhorizontal and vertical directions and in an inter-frame manner.

However, according to the related art disclosed in the '471 documentdescribed above, when a pixel defect occurs in an image pickup elementthat forms an image pickup system for obtaining lights passing throughdifferent pupil areas, a proper image may not always be obtained.

In other words, in the '471 document, even though an image in which afocus position is changed after image pickup may be obtained, a methodof obtaining an image excluding an influence of a defected portion in acase when a pixel defect is present is not disclosed. Further, in theTakemura article, since the interpolation processing is performed on thebasis of simple image correlation, appropriate interpolation may not beperformed on the image pickup element that forms an image pickup systemof the '471 document.

SUMMARY OF THE INVENTION

It is an aspect of the invention to provide an image pickup apparatusthat is capable of obtaining a high quality image, in which influence bythe pixel defect is reduced, even when using an image pickup elementthat is capable of obtaining information on lights passing throughdifferent pupil areas.

In order to achieve the aspect of the invention, an image pickupapparatus of the invention, which includes a photographing opticalsystem having a photographing lens and an image pickup element thatphotoelectrically converts an optical image of an object arrivingthrough the photographing lens to output an image signal, comprises amemory unit that stores information on a pixel defect of the imagepickup element and information for determining an angle of incidence ofthe optical image arriving at each pixel of the image pickup element, animage generation position setting unit that sets an image generationposition where a re-formed image is generated from the image signal, apupil division unit that restricts light of the optical image of theobject arriving at each pixel of the image pickup element to a lightfrom a specific pupil area of the photographing lens, an image shiftunit that determines a shift amount of the image signal corresponding tothe image generation position for every pupil area to shift the imagesignal, on the basis of the image generation position set by the imagegeneration position setting unit and information for determining theangle of incidence of the optical image arriving at each pixel of theimage pickup element, a defect correction unit that corrects an imagesignal of a defective image using an image signal of a pixel other thanthe defective pixel, obtained by the image shift unit, in accordancewith the information on the pixel defect, and an image generation unitthat generates an image corresponding to the image generation positionfrom the image signal, which is corrected by the defect correction unit.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a system configuration of aphotographing apparatus according to an embodiment of the invention.

FIGS. 2A, 2B, and 2C are schematic diagrams of an image pickup system ofa photographing apparatus according to a first embodiment of theinvention.

FIGS. 3A, 3B, 3C, 3D, and 3E are flowcharts of an image re-formingoperation according to the first embodiment of the invention.

FIGS. 4A, 4B, and 4C are views conceptually illustrating focusdetermination in an operation of contrast AF.

FIGS. 5A, 5B, 5C, and 5D are views schematically illustratingre-formation of an image.

FIGS. 6A, 6B, and 6C are views schematically illustrating imagegeneration in an image pickup system.

FIGS. 7A, 7B, and 7C are flowcharts of a re-forming operation of animage according to a second embodiment.

DESCRIPTION OF THE EMBODIMENTS

Exemplary embodiments of the present invention will be described indetail in accordance with the accompanying drawings.

First Embodiment

Hereafter, a photographing apparatus according to a first embodiment ofthe invention will be described with reference to FIGS. 1 to 6C.

FIG. 1 is a block diagram illustrating a configuration of a camerasystem including a digital camera and a lens as a photographingapparatus according to the invention. The camera system includes animage pickup system for obtaining lights passing through different pupilareas and has a function of collecting an image defect, which is anobject of the invention.

The camera system that includes a camera 101 and a lens 102(photographing lens) includes an image pickup system, an imageprocessing system, a recording and reproducing system, and a controlsystem. The image pickup system includes a photographing optical system103 and an image pickup element 106. The image processing systemincludes an image processing unit 107. The recording and reproducingsystem includes a memory unit 108 and a display unit 109. The controlsystem includes a camera system control unit 105, an operation detectionunit 110, a lens system control unit 112, and a lens drive unit 113. Thelens drive unit 113 may drive a focus lens, a vibration correction lens,and an iris included in the photographing optical system 103.

The image pickup system is an optical processing system that focuseslight from an object (optical image) on an image pickup plane of theimage pickup element 106 through the photographing optical system 103.Micro lenses are arranged in a matrix form on a surface (light receivingsurface) of the image pickup element 106 to form a micro lens array(hereafter, abbreviated as MLA). In this embodiment, the MLA functionsas a pupil division unit that divides a plurality of pixels into aplurality of pixel groups in accordance with the respective microlenses. Details of the function and arrangement of the MLA will bedescribed below with reference to FIGS. 2A to 2C. However, a focusevaluation value/an appropriate exposure amount from the image pickupelement 106 may be obtained by providing the pupil division unit, sothat the photographing optical system 103 is appropriately adjusted onthe basis of the obtained information. By doing this, it is possible toexpose an object light with an appropriate light intensity to the imagepickup element 106 and form an object image in the vicinity of the imagepickup element 106.

The image processing unit 107 includes an A/D converter, a white balancecircuit, a gamma correction circuit, and an interpolation operationcircuit therein and creates a recording image by the image processing.Further, the image processing unit 107 may include an image shift unit,an image generation unit, a contrast evaluation unit, and a correlationcalculation unit, which are main parts of this embodiment. In thisembodiment, the above-mentioned elements are configured as a controlprogram under the control of the camera system.

The memory unit 108 includes, not only a storage unit that actuallystores data, but also a processing circuit that is required to recorddata. The memory unit 108 outputs an image to a recording unit andgenerates and stores an image to be output to the display unit 9.Further, the memory unit 108 compresses an image, a moving image, or anaudio using a predetermined method.

The camera system control unit 105 generates and outputs a timing signalat the time of image pickup and controls the image pickup system, theimage processing system, and the recording and reproducing system inresponse to external operation. For example, the operation detectionunit 110 detects depression of a shutter release button, which is notillustrated, to control drive of the image pickup element 106(photoelectric conversion), operation of the image processing unit 107,and compression processing of the memory unit 108. Further, the camerasystem control unit 105 also controls the state of segments of aninformation display device that displays information on a liquid crystalmonitor by the display unit 109.

An adjustment operation of the optical system by the control system willbe described. The camera system control unit 105 is connected with theimage processing unit 107 to calculate a focus position and an irisposition suitable for a photographing condition, on the basis of asignal from the image pickup element 106. The camera system control unit105 transmits an instruction to the lens system control unit 112 throughan electrical connection 111 and the lens system control unit 112appropriately controls the lens drive unit 113 in accordance with theinstruction. Further, a vibration detection sensor, which is notillustrated, is connected to the lens system control unit 112 toappropriately control a vibration correction lens through the lens driveunit 113, on the basis of a signal of the vibration detection sensor ina vibration correction mode.

FIGS. 2A to 2C are diagrams illustrating main parts of the photographingoptical system in the present embodiment of the invention. In FIGS. 2Ato 2C, the same elements as those in FIG. 1 are denoted with the samereference numerals.

In order to apply the present invention to the camera system of FIG. 1,a configuration that obtains information of an angle in addition to aposition of a ray of light, which is called light field information, isrequired. In the present embodiment, in order to obtain the angleinformation, the MLA is disposed in the vicinity of a focal plane of thephotographing optical system 103 and a plurality of pixels correspond toone lens that forms the MLA.

FIG. 2A is a view schematically illustrating a correspondencerelationship between the image pickup element 106 and the MLA 200. FIG.2B is a view schematically illustrating a correspondence between a pixelof the image pickup element and the MLA. FIG. 2C is a conceptual viewillustrating that a pixel provided below the MLA is associated with aspecific pupil area by the MLA 200.

As illustrated in FIG. 2A, the MLA 200 is provided on the image pickupelement 106 and a front principal point of the MLA 200 is positioned soas to be in the vicinity of the focal plane of the photographing opticalsystem 103. FIG. 2A illustrates a side view of the photographingapparatus and a front view of the MLA 200. The lenses of the MLA 200 arearranged so as to cover the pixels on the image pickup element 106. InFIG. 2A, even though the micro lenses that form the MLA 200 are enlargedin order to be more recognizable, actually, each micro lens has a sizeseveral times larger than the pixel. The actual size will be describedwith reference to FIG. 2B.

FIG. 2B is a view that enlarges a part of the front view of the MLA 200of FIG. 2A. Frames having a matrix form illustrated in FIG. 2Billustrate pixels of the image pickup element 106. In the meantime, themicro lenses that form the MLA 200 are represented by thick circles 220a, 220 b, 220 c, and 220 d. As is apparent from FIG. 2B, a plurality ofpixels are allocated to one micro lens and, in an example of FIG. 2B,twenty-five pixels (5 rows×5 columns) form one group correspondingly toone micro lens. In other words, a size of each of the micro lenses is 5times×5 times of a size of the pixel.

FIG. 2C is a view illustrating a cross section of one micro lens whenthe image pickup element 106 is cut at such a plane that an optical axisof the micro lens is included, and a longitudinal direction(X-direction) of the image pickup element becomes a horizontal directionof the drawing. Reference numerals 221, 222, 223, 224, and 225 of FIG.2C denote pixels (one photoelectric converting unit) of the image pickupelement 106. In meantime, a drawing in the upper portion of FIG. 2Cillustrates an exit pupil plane of the photographing optical system 3.Actually, if it is matched with a drawing of the image pickup elementillustrated in a lower portion of FIG. 2C, in terms of direction, theexit pupil plane (X-Y plane) is parallel to a vertical direction (ydirection) of the drawing of FIG. 2C. However, for the convenience ofthe description, the projective direction is changed. Further, in FIG.2C, for the sake of simplicity of the description, one-dimensionalprojection/signal processing will be described. In other words, pupildivision is of one dimension of only 231 to 235 and a correspondingpixel arrangement is also, for example, of one dimension of only 321 ato 325 a of FIG. 2B. This assumption is also applied to the descriptionof FIGS. 5A to 5D. In an actual apparatus, this may be easily expandedto two dimensions.

The pixels 221, 222, 223, 224, and 225 of FIG. 2C may have a positionalrelationship to correspond to 221 a, 222 a, 223 a, 224 a, and 225 a ofFIG. 2B, respectively. As illustrated in FIG. 2C, the pixels aredesigned so as to be conjugate, by the micro lens 200, with a specificarea on the exit pupil plane of the photographing optical system 103. Inan example of FIG. 2C, the pixel 221 corresponds to the area 231, thepixel 222 corresponds to the area 232, the pixel 223 corresponds to thearea 233, the pixel 224 corresponds to the area 234, and the pixel 225corresponds to the area 235. In other words, only light passing throughthe area 231 on the exit pupil of the photographing optical system 103arrives at the pixel 221. The other pixels are the same as those above.As a result, it is possible to obtain information of an angle on thebasis of a relationship between a passage area on the pupil plane andthe position on the image pickup element 106.

In the following description, for the sake of simplicity, symbols areintroduced. As illustrated in FIG. 2C, a pixel pitch of the image pickupelement 106 is denoted with x and an angular resolution is denoted withΔθ. Further, a division number of the angle is denoted with N (in theexample of FIG. 2C, N=5). The pixel pitch is determined depending on ashape of the image pickup element 106 and Δθ is determined by a rangewhere an angle of a ray of light is obtained and the division number ofthe angle. That is, the parameters are determined by only the physicalstructure of the image pickup system (structure of the image pickupelement 106 and the MLA 200).

Using the photographing optical system illustrated in the presentembodiment, a processing that obtains an image whose defect isappropriately processed from the signal of the image pickup element 106will be described with reference to FIGS. 3A to 3E, 4A to 4C, and 5A to5D.

FIGS. 3A to 3E are flowcharts of a re-forming operation of an image inthe camera system of the present embodiment. FIG. 3A is a flowchartillustrating an overall image re-forming operation, FIG. 3B is aflowchart illustrating an operation of a defect correction unit, FIG. 3Cis a flowchart illustrating an operation of an image shift unit, FIG. 3Dis a flowchart illustrating an operation of an image generation unit,and FIG. 3E is a flowchart illustrating an operation of a correlationcalculation unit.

The embodiment will be described from FIG. 3A in the order of steps.Step S301 indicates starting of the image obtaining operation. Forexample, the operation detection unit 110 illustrated in FIG. 1 detectsa specific operation from a photographer (for example, depression of arelease button).

In step S302, after exposing the image pickup element 106 for anappropriate time, an image pickup signal is read and A/D converted toobtain an image signal. An image pickup condition, such as an exposuretime, is set by the camera system control unit 105 in accordance withinformation from a photometry unit, which is not illustrated.

In step S303, the correlation calculation unit is operated to obtain acorrelation calculation result. Information on the focus evaluationvalue is obtained from the correlation calculation unit. Details of theoperation will be described below with reference to FIG. 3E.

In step S304, a focus position (image generation position) is determinedfor every set area (which will be described below in the description ofthe correlation calculation unit, but corresponds to an evaluation framein step S362). A position where a best focus evaluation value isobtained from the correlation calculation unit, which will be describedbelow, is determined and set as the focus position. Here, the meaning of“best” is a state where a value by a correlation calculation equation instep S366, which will be described below, is small.

However, herein, the focus position indicates a relative defocusing froma current focus position. In other words, if a focus position of anobject, which is in focus at the current focus position, is zero, afocus position before or after the current focus position is obtained asa value with a plus or minus symbol. Further, a focus position isobtained, not a depth on the object side, but as a position of the focalplane on the image plane side.

In step S305, the defect correction unit is operated to obtain a resultof the correction. Details of the operation of the defect correctionunit will be described with reference to the flowchart of FIG. 3B.

In step S306, the image shift unit is operated to obtain a result of theimage shift for generating an image. In this case, several methods areconsidered for the image generation position. For example, a method thatdefines a focus position by considering the focus positions of the areasdetermined in step S304 and the object recognition result together isconsidered. Accordingly, it is possible to focus an object recognized asan object. As another method, it is possible to assign a positiondesignated by a user. By doing this, it is possible to achieve manualfocus. Details of the operation of the image shift unit will bedescribed below with reference to the flowchart of FIG. 3C.

In step S307, the image generation unit is operated to obtain a resultof image generation. Details of the operation of the image generationunit will be described below with reference to the flowchart of FIG. 3D.

In step S308, a processing required for recording, such as conversion orcompression into a recording image format, is performed on an generatedimage signal, and then, the image signal is recorded in the memory unit108.

In step S309, a series of operations from the obtaining of the imagesignal to the recording of the image signal are completed.

Details of the operation of the defect correction unit will be describedwith reference to the flowchart of FIG. 3B. Step S371 indicates startingof the operation of the defect correction unit.

In step S372, defect information and a constant number d_(max) areobtained from the memory unit 108. Here, d_(max) is a value indicating arange of the image generation position where an appropriately re-formedimage is obtained by the image generation unit and given by thefollowing Equation.

$\begin{matrix}{d_{\max} = \frac{N_{\theta}\Delta\; x}{\tan\;\Delta\;\theta}} & (1)\end{matrix}$

In this case, d_(max) denotes a threshold value that normally forms animage, N_(θ) denotes a pupil division number, Δx denotes a pixel pitchof the image pickup element 106, and Δθ denotes an angle resolution. Asdescribed with reference to FIGS. 2A to 2C, these values are determinedby the physical structure of the image pickup apparatus irrespectivelyof the photographing condition. Therefore, a value of d_(max), which iscalculated in advance, is stored in the memory unit 108 and then read tobe used in steps S372 and S373.

The defect information refers to information on pixel defect and,specifically, to information concerning an address of a pixel in whichflared highlights (a phenomenon where a luminance value is a luminancesaturation value) or blocked up shadows (a phenomenon where a luminancevalue is obtained as zero) occur. As described above, a pixel, whichoutputs an abnormal luminance value generated due to the pixel defect,is referred to as a defective image. Since the defective image in whichthe flared highlights or blocked up shadows are generated regardless ofthe object may be not used for development, a processing has beenusually performed by a method disclosed in the Takemura article. In thisembodiment, these processings are performed in steps S374 to S380, ifnecessary. Further, since such a pixel is small and does not move, thedefect information may be stored in the memory unit 108 and then readand used in steps S372 and S373.

Steps S374 to S380 are loop processings and processings in steps S374 toS380 are performed on all defective images.

Step S375 compares the absolute value of the focus position determinedin step S304 and an absolute value (threshold value) of d_(max). If theabsolute value of the focus position is equal to or lower than thethreshold value, which is the absolute value of d_(max), the sequenceproceeds to step S377 and, otherwise, the sequence proceeds to stepS376.

An interpolation processing of the defective image is performed by aknown technology in step S376. In other words, before performing aspecific processing on image signals obtained from pixels other than thedefective image, correlation of the defective image in a verticaldirection or a horizontal direction is calculated, and the interpolationprocessing is performed in a direction having a higher correlation. InFIG. 3B, the interpolation processing is illustrated as “interpolationprocessing on the image pickup image (interpolation processing that usesan image signal other than an output of the image shift unit).” Theinterpolation may be represented by the following equation.X _(i,j)=(X _(i−1,j) +X _(i+1,j))/2 if |X _(i−1,j) −X _(i+1,j) |≦|X_(i,j−1) −X _(i,j+1)|X _(i,) j=(X _(i,j−1) +X _(i,j+1))/2 if |X _(i−1,j) −X _(i+1,j) |>|X_(i,j−1) −X _(i,j+1)|  (2)

In Equation 2, a technical idea of a very basic interpolation isrepresented. Here, addresses of the defective images are denoted with iand j and a luminance value at the addresses i and j is denoted withX_(i,j). In Equation 2, |X_(i−1,j)−X_(i+1,j)| is compared with|X_(i,j−1)−X_(i,j+1)|, which corresponds to the determination of adirection having a higher correlation in the horizontal direction andthe vertical direction. An object image in a natural image has acharacteristic different from a random pattern. Even though theluminance is sharply changed in an edge direction, in many cases, acomparatively flat luminance distribution is obtained in a directionthat does not have an edge. Using this characteristic, under theassumption that the luminance distribution is flat in a direction where|X_(i−1,j)−X_(i+1,j)| and |X_(i,j−1)−X_(i,j+1)| are small, theinterpolation processing is performed. Further, even though a moreadvanced processing method is suggested, only fundamentals of atechnical idea are suggested here.

In step S376, since the defect is corrected by the related art, apositive effect obtained by using the present invention may be notachieved. However, in step S375, an effect may be obtained bydetermining the absolute value of the focus position and the absolutevalue of d_(max). Since it is understood that the defective image has anobject image, which is defocused beyond d_(max) (a case when it isdetermined to be “NO” in S375), even though the image is re-formed by asimple interpolation processing, the defective image does not affect theimage. That is, in the operation of the image generation unit, which isperformed in step S307, it is understood that the defective image doesnot cause a very serious affect (occurrence of ringing). Sincedefocusing beyond a range d_(max) within which a re-formed image can beobtained readily results in that the re-formed image will be blurred, animage is obtained in a state where information on a radio-frequencyobject image is missed. In this phenomenon, it is possible to obtain aconsiderably good interpolation value even by the simple interpolationprocessing indicated by Equation 2 (here, luminance information to beobtained at X_(i,j) spreads due to blurring to leak into peripheralpixels, which is synonymous with the radio frequency component beingcut).

In step S377, the image shift unit is operated so as to be focused at aposition of the defective image on which a current processing isperformed to obtain a result. In other words, the image signal is in astate where the image shift on re-formation planes of FIGS. 5A to 5D iscompleted, but a processing that performs addition in a verticaldirection to obtain Si is not performed. Details of the operation of theimage shift unit will be described below with reference to FIG. 3C.

In step S378, the interpolation processing is performed on the basis ofthe result of the image shift obtained from step S377, which isillustrated as “interpolation processing on the shift image”), in FIG.3B. Specifically, as a result of the operation of the image shift unitin step S377, the image is focused in the position of the defectiveimage of the image on which the current processing is performed. In thisstate, as it is well known, the same information may be obtained,regardless of the pupil position to be passed. This is because a ray oflight emitted from one point on the same object is focused at one pointin the developed image. In this case, it is proper to performinterpolation from information that passes through apparently differentpupil areas. It may easily be understood if the correlation calculation,which will be described below with reference to FIGS. 4A to 4C, isreferred to (even though the simple description will be made withreference to FIGS. 4A to 4C herein, a detailed description of FIGS. 4Ato 4C will be made below). In FIGS. 4A to 4C, FIG. 4B illustrates afocused state. In this case, distributions A_(i) and B_(i) of theluminance values of the information that passes through the differentpupil areas have a similar shape. For example, if a pixel A_(k) is adefective image, a proper processing is performed by copying a luminancevalue of a corresponding pixel of B_(i) in FIG. 4B. For the sake ofsimplicity, FIGS. 4A to 4C illustrate information of two pupil passageareas. If the pupil division number is large, since more informationfrom a pixel that is not defective may be obtained, another appropriateinterpolation method may be used instead of copying the luminance value.As an example, it is considered to use an average value or a dispersionof the luminance values of normal pixels.

By the processings to step S379, the interpolation processing isperformed using information on a focused plane (as described above, thesame information may be obtained regardless of the passed pupil area) orinformation having good correlation (=high similarity). Therefore, anappropriate interpolation processing is performed for the defectivepixel. As a result, it is possible to obtain a high quality image evenin the re-formation of the image.

In step S380, the sequence returns to step S305, which is a caller ofthe defect correction routine.

Details of the operation of the image shift unit will be described withreference to FIG. 3C. Step S321 indicates starting of the operation ofthe image shift unit.

Steps S322 to S326 are loop processings. In step S322, a loop iscalculated as many as the number corresponding to the pupil divisionnumber. For example, in the example illustrated in FIGS. 2A to 2C, sincethe exit pupil of the photographing optical system 103 is divided intotwenty-five parts, calculation is performed according to twenty-fivepupil positions. As will be described below with reference to FIGS. 5Ato 5D, if an angle of incidence is different even on the samere-formation plane (this is substantially synonymous with a passed pupilarea being different in a case when there is enough distance to the exitpupil), an amount by which an image is shifted for re-formation isvaried. The loop is a loop for appropriately reflecting the abovedescription.

In step S323, based on data from step S324, an image shift amount in thepupil area corresponding to the evaluation position is calculated. Instep S324, a correspondence relationship between the pixels and the MLAis stored and information indicating the pupil area from which pupilarea each pixel receives a ray of light is stored.

In step S325, on the basis of the information obtained in step S323,pixels obtaining the rays of light having the same angle of incidence(obtaining the rays of light from the same pupil area) are shifted. Thepixels obtaining the rays of light having the same angle of incidenceare, for example, 225 a and 225 b of FIGS. 2A to 2C. Such pixels arepresent as many as the number of micro lenses that form the MLA 200.

The image shift unit is described in more detail with reference to FIGS.5A to 5D (which will be described below).

In step S327, the sequence returns to step S304, which is a caller ofthe image shift unit or step S377.

Details of the operation of the image generation unit will be describedwith reference to FIG. 3B.

Step S331 indicates starting of the operation of the image generationunit.

In step S332, an area where data of an addition result in step S335 isstored is initialized (filled with zero). In this case, the data areapreferably has a size corresponding to the number of MLAs 200. Further,a gradation of data may be enough if the product of the gradation oforiginal data and the pupil division number can be stored. For example,when the data is eight bits and the pupil division number istwenty-five, if the gradation of data is thirteen bits (>8 bits+log₂25), there is no need to consider the overflow of data.

Steps S333 to S338 are loop processings. In step S333, loop calculationis performed as many times as the number of micro lenses that form theMLA 200. For example, in the examples illustrated in FIGS. 2A to 2C, thenumber of pixels of the image pickup element÷twenty-five (pupil divisionnumber) provides the number of micro lenses.

Steps S334 to S337 are loop processings. In step 334, loop calculationis performed as many times as the pupil division number. For example, inthe examples illustrated in FIGS. 2A to 2C, since the exit pupil isdivided into twenty-five, lights from twenty-five pupil positions areprocessed.

In step S335, it is determined whether a pupil area is to be added. Inother words, by changing the area to be added in accordance with thesetting of the user, an intended image is provided. Generally, if thepupil area to be added is increased, an image may have a small depth offocus due to an excellent S/N, and if the pupil area to be added isdecreased, an image may have a large depth of focus.

In step S336, addition is performed. If the shift amount is not integertimes as long as the pixel, in the addition of S336, light is properlydivided to be added (for example, appropriately added in accordance withthe overlapping area).

The image generation unit will be more specifically described withreference to FIGS. 5A to 5D (will be described below).

In step S339, the sequence returns to step S307, which is the caller ofthe image generation unit.

Details of the operation of the correlation calculation unit will bedescribed with reference to FIG. 3E.

Step S361 indicates starting of an operation of the correlationcalculation unit.

In step S362, the number of evaluation points that are subjected to theevaluation and a size of the evaluation frame (for example, a framehaving the evaluation point at the center thereof) are set. Theevaluation point is appropriately set in accordance with thephotographing condition or a type of lens 102. Further, if theevaluation frame is too large, perspective conflict in which images ofobjects having different distances simultaneously are evaluated mayoccur. Here, the evaluation frame is preferably as small as is possiblewithin a range where the correlation may be appropriately calculated inspite of the noise. By doing this, the focus position is appropriatelydetected when the defect is corrected.

Steps S363 to S369 are loop processings. In step S363, arithmetic isrepeatedly performed so as to obtain an evaluation value correspondingto the number of evaluation points determined in step S362.

Steps S364 to S367 are loop processings. In step S364, the correlationcalculation is performed for the pixel included in the evaluation framedetermined in step S362.

In step S365, it is determined that the pixel Ai or Bi is defective.Here, A_(i) denotes luminance of an i-th pixel corresponding to aspecific passed pupil area. B_(i) denotes luminance of an i-th pixelcorresponding to a passed pupil area, which is different from A_(i). Forexample, in FIGS. 2A to 2C, A_(i) may be formed by arranging only pixelscorresponding to the pixel 222 and B_(i) may be formed by arranging onlypixels corresponding to the pixel 224. The pupil area may be determineddepending on a reference length or a vignetting state of the pupil planeto select the pixel corresponding to the determined pupil area.

In step S365, if the pixel is determined to be defective, this pixel isnot proper to use for the correlation calculation, so that the sequenceproceeds to step S367. If the pixel is not defective, the sequenceproceeds to step S366. The correlation calculation may be performed, forexample, by Σ|A_(i)−B_(i)| as represented in step S366. Further, in thisembodiment, the correlation calculation is used so as to find the focusposition of the defective pixel and, thus, the equation of thecorrelation calculation represented herein is only illustrative, and notthe essence of the invention.

By setting as described above, the correlation of the images that passthrough different pupil areas may be calculated and the evaluationamount may be obtained on the basis of a phase different AF. In stepS368, the obtained correlation value is stored as the evaluation amount.

In the evaluation equation of Σ|A_(i)−B_(i)| described above, a portionwhere the correlation value becomes smaller corresponds to a portionhaving a good focus state. Here, the correlation calculation isperformed using a method that adds a differential absolute value.However, the correlation calculation may be performed using othercalculation methods, such as a method that adds a maximum value, amethod that adds a minimum value, or a method that adds a differentialsquare value.

In step S369, the sequence returns to step S307, which is a caller ofthe correlation calculation unit.

An example of distribution of the luminance values corresponding to theevaluation in the correlation calculation unit described here isillustrated in FIG. 4A to 4C.

FIGS. 4A to 4C illustrate the distribution of the luminance values whenthe focus position is changed in this order. The horizontal axisindicates a pixel position and the vertical axis indicates the luminancevalue. As described above, in FIGS. 2A to 2C, Ai is formed by arrangingonly pixels corresponding to the pixel 222 and Bi is formed by arrangingonly pixels corresponding to the pixel 224. In the correlationcalculation unit, for example, the evaluation is performed by thefollowing equation.C(p)=by Σ|A _(i-q) −B _(i-q)  (3)where summation is performed over q from 1 to the number of evaluationpixels, while excluding a defective pixel.

Here, C(p) denotes an evaluation amount of the correlation and is acorrelation evaluation value of the correlation arithmetic method calledSAD (sum of absolute difference), which is an example of the correlationevaluation method. Therefore, the evaluation may be performed by usingother correlation arithmetic methods, for example, SSD (sum of squareddifference).

In FIG. 3E, q represents a pixel that is a target of the repeatingprocessing in the loop from step S364 to step S367. According toEquation 3, correlation of A_(i) and B_(i) is calculated, while shiftingthe corresponding pixels. As illustrated in FIGS. 5A to 5D, this casemay correspond to a case when the correlation of pixel rows is obtainedby deviating the adjacent pixel column, which is represented in parallelin the vertical direction in FIGS. 4A to 4C. In other words, in FIG. 4A,a peak of A_(i) is positioned further left than a peak of B_(i). In FIG.4B, the peak of A_(i) matches with the peak of B_(i). In FIG. 4C, thepeak of A_(i) is positioned further right than the peak B_(i), which isa behavior of an image passing through the different pupil areas whenthe focus position is varied.

An in-focus state is illustrated in FIG. 4B. However, in this case, thesame information may be obtained regardless of the passed pupil area,which corresponds to a case when the A_(i), and B_(i) overlap in FIG.4B. Ai and Bi substantially match with each other. Non-matched partsdepend on a noise component and a difference of the diffusioncharacteristic of the light of the object.

If it is considered that a specific pixel A_(k) is defective, in FIGS.4A to 4C, a defect of A_(k) may be processed using a value of a pixelB_(j) corresponding to A_(k) in FIG. 4B. Here, it should be noted thatBj is not necessarily a pixel adjacent to the Ak in the photographedstate. As is apparent from Equation 3, if q=0, it is assumed that B_(i)is present in the vicinity of A_(k). However, in other cases, B_(j) isnot necessarily an adjacent pixel to the A_(k). In the meantime, whenthe image is re-formed so as to be focused, B_(j) is present in thevicinity of A_(k). Such a pixel is specified to be used in the defectcorrection unit. In the present embodiment, there is no need to specifyand to record the pixel. However, if a focusing position is specified, apixel corresponding to A_(k) is naturally determined, so that thefocusing position is stored.

Next, using FIGS. 5A to 5D, the image shift and the image generation areschematically illustrated and a usability of contrast calculation byre-forming an image will be described.

In FIGS. 5A to 5D, first, FIG. 5B illustrates a surface in which animage is obtained by the image pickup element 106, which is actuallyprovided thereon, FIG. 5A illustrates a re-formation plane closer to theobject side more than in FIG. 5B (referred to as a re-formation plane1), FIG. 5C illustrates a re-formation plane (referred to as are-formation plane 2), which is separated away from the object side morethan in FIG. 5B. As described above, in FIGS. 5A to 5D, for the sake ofa clear and brief description, the pupil division direction and thepixel arrangement are one dimensional.

In FIG. 5B, X_(1,i), X_(2,i), X_(3,i), X_(4,i), and X_(5,i) indicatedata obtained as light passes through the pupil areas 1, 2, 3, 4, and 5to arrive at the micro lens X_(i). In other words, a front part of thelower suffix indicates the passed pupil area and a rear part of thelower suffix indicates a number of the micro lens. In the relationshipwith a physical position, X_(1,i) indicates data obtained from area 221of FIG. 2C and X_(2,i) indicates data obtained from area 222 of FIG. 2C.Further, lower suffixes 3, 4, and 5 correspond to the areas 223, 224,and 225.

In order to generate an image on an obtaining plane, as illustrated inFIG. 5B, data arriving at the micro lens Xi may be added. Specifically,an integral value in an angular direction of the light arriving at Ximay be obtained by Si=X_(1,i)+X_(2,i)+X_(3,i)+X_(4,i)+X_(5,i). Byperforming the above operation for all of the micro lenses, an image,which is similar to a normal camera, is created.

Next, a method of generating an image on the re-formation plane 1 willbe considered. As described in FIGS. 2A to 2C, in the photographingoptical system of this embodiment, the light that arrives at the pixelsis limited in a specific pupil area, and thus, a known angle ofincidence is used. The positions of the pixels on the re-formation planeare re-formed according to the angles. Specifically, like X_(1,i), ifthe suffix of the pupil area is 1, in FIG. 5D, the light arrives at anangle denoted by reference numeral 541. Hereafter, the suffixes 2, 3, 4,and 5 of the pupil areas correspond to reference numerals 542, 543, 544,and 545, respectively. At this time, the light that arrives at the microlens X_(i) on the re-formation plane 1 is dispersed into X_(i−2) toX_(i+2) (one dimension) so as to arrive at the obtaining plane. Morespecifically, the light is dispersed into X_(1,i−2), X_(2,i−1), X_(3,i),X_(4,i+1), and X_(5,i+2). As to the micro lenses other than X_(i), itcan be also understood that it is required to re-form an image at there-formation plane 1 only to shift and to add the image in accordancewith the angle of incidence. In order to generate an image at there-formation plane 1, a micro lens with a suffix 1 of the pupil area isshifted by two pixels to the right, a micro lens with a suffix 2 of thepupil area is shifted by one pixel to the right, and a micro lens with asuffix 3 of the pupil area is not shifted. Further, a micro lens with asuffix 4 of the pupil area is shifted by one pixel to the left and amicro lens with a suffix 5 of the pupil area is shifted by two pixels tothe left. Therefore, the micro lenses may be shifted correspondingly tothe angles of incidence. Thereafter, data at the re-formation plane 1may be obtained by the addition in the vertical direction in FIG. 5A.Specifically, at the re-formation plane 1 ofSi=X_(1,i−2)+X_(2,i−1)+X_(3,i)+X_(4,i+1)+X_(5,i+2), an integral value inthe angle direction of the light which arrives at X_(i) may be obtained.Accordingly, an image at the re-formation plane 1 is obtained.

Here, at the re-formation plane 1, if it is considered that X_(i) has abright point, the light is dispersed onto X_(1,i−2), X_(2,i−1), X_(3,i),X_(4,i+1), and X_(5,i+2) at the obtaining plane to be in a blurringstate. However, if the image is generated at the re-formation plane 1,the bright point is generated on Xi again so that an image having a highcontrast is obtained. In other words, by re-reforming the image tocalculate the contrast, so-called contrast AF may be performed.

Further, as known from FIG. 5C, also at a re-formation plane 2, an imagemay be generated by just the same manner as at the re-formation plane.If the direction in which the re-formation plane is positioned is varied(which means an opposite side to the object), a shift direction onlyneeds to be inversed.

Referring to FIGS. 6A to 6C, an example that re-forms an image at avirtual focal plane in the image pickup system of FIGS. 2A to 2C will bedescribed.

FIGS. 6A to 6C are views schematically illustrating a state when thelight from an object is focused on the image pickup element 106. FIG. 6Acorresponds to the focus state of the optical system described withreference to FIGS. 2A to 2C, and is an example that the MLA 200 isdisposed in the vicinity of the focal plane of the photographing opticalsystem 103. FIG. 6B is a view illustrating the focusing state when theMLA 200 is disposed closer to the object than the focal plane of thephotographing optical system 103. FIG. 6C is a view illustrating thefocusing state when the MLA 200 is disposed farther from the object thanthe focal plane of the photographing optical system 103.

In FIGS. 6A to 6C, reference numeral 106 denotes the image pickupelement, reference numeral 200 denotes the MLA, reference numerals 231to 235 denote the pupil areas used in FIGS. 2A to 2C, reference numeral651 denotes an object plane, reference numerals 651 a and 651 b denoteappropriate points on the object, and reference numeral 652 denotes apupil plane of the photographing optical system. Further, referencenumerals 661, 662, 671, 672, 673, 681, 682, 683, and 684 denote specificmicro lenses on the MAL. Reference numeral 6 a illustrated in FIGS. 6Band 6C denotes an image pickup element disposed on the virtual focalplane and reference numeral 200 a denotes an MLA disposed on the virtualfocal plane, which are referred to so as to clarify the correspondencerelationship with FIG. 6A. Further, the light that is emitted from thepoint 651 a on the object and passes through areas 231 and 233 on thepupil plane is represented by a solid line, and the light that isemitted from the point 651 b on the object and passes through areas 231and 233 on the pupil plane is represented by a broken line.

In the example of FIG. 6A, as described with reference to FIGS. 2A to2C, the MLA 200 is disposed in the vicinity of the focal plane of thephotographing optical system 103 so that the image pickup element 106 isconjugate with the pupil plane 652 of the photographing optical system.Further, the object plane 651 is conjugate with the MLA 200. Therefore,the light emitted from the point 651 a on the object arrives at themicro lens 661 and the light emitted from the point 651 b arrives at themicro lens 662. Further, the lights that pass through the areas 231 to235 arrive at corresponding pixels provided below the micro lenses.

In the example of FIG. 6B, the micro lens 200 allows the light from thephotographing optical system 3 to be focused and an image pickup element106 is provided on the focal plane. With this arrangement, the objectplane 651 is conjugate with the image pickup element 106. The light thatis emitted from the point 651 a on the object and passes through an area231 on the pupil plane arrives at the micro lens 671 and the light thatis emitted from the point 651 a on the object and passes through an area233 on the pupil plane arrives at the micro lens 672. The light that isemitted from the point 651 b on the object and passes through an area231 on the pupil plane arrives at the micro lens 672 and the light thatis emitted from the point 651 b on the object and passes through an area233 on the pupil plane arrives at the micro lens 673. Further, the lightthat passes through each micro lens arrives at corresponding pixelsprovided below the micro lens. Therefore, the lights are focused ondifferent positions of the image pickup element by the points on theobject and the passing area on the pupil area. With the re-arrangementin a position on the virtual image pickup plane 106 a, the sameinformation as shown in FIG. 6A may be obtained. In other words,information on a passed pupil area (angle of incidence) and a positionon the image pickup element may be obtained and a function as a pupildivision unit may be achieved.

In an example of FIG. 6C, the micro lens 200 allows the light from thephotographing optical system 3 to be refocused (refocusing means thatfocuses again a focused light, which is in a spread state) and an imagepickup element 106 is provided on the focal plane. With thisarrangement, the object plane 651 is conjugate with the image pickupelement 106. The light that is emitted from the point 651 a on theobject and passes through an area 231 on the pupil plane arrives at themicro lens 682 and the light that is emitted from the point 651 a on theobject and passes through an area 233 on the pupil plane arrives at themicro lens 681. The light that is emitted from the point 651 b on theobject and passes through an area 231 on the pupil plane arrives at themicro lens 684 and the light that is emitted from the point 651 b on theobject and passes through an area 233 on the pupil plane arrives at themicro lens 683. Further, the lights that pass through the micro lensesarrive at corresponding pixels provided below the micro lenses. Similarto that shown in FIG. 6B, with the re-arrangement in a position on thevirtual image pickup plane 106 a, the same information as shown in FIG.6A may be obtained. In other words, information on a passed pupil area(angle of incidence) and a position on the image pickup element may beobtained, and a function as a pupil division unit may be achieved.

In FIGS. 6A to 6C, an example that uses the MLA (phase modulationelement) as the pupil division unit to obtain the position informationand angle information is illustrated. However, if the positioninformation and the angle information (equivalence to limiting thepassing area of the pupil) are obtained, other optical configurationsmay be used. For example, a method of inserting a mask with anappropriate pattern thereon (gain modulation element) in an opticalpassage of the photographing optical system may be used.

As described above, according to the present embodiment, even though animage pickup element having a pixel defect is used, on the basis of theinformation of the light that passes through different pupil areas inthe image pickup element, it is possible to obtain a high quality imagein which the influence of the pixel defect is reduced.

Even though the exemplary embodiment of the present invention has beendescribed above, the present invention is not limited to the exemplaryembodiment, but may be modified or changed without departing from thegist of the invention.

Second Embodiment

Hereafter, referring to FIGS. 7A to 7C, a configuration of a camerasystem according to a second embodiment of the present invention will bedescribed. In this embodiment, since the camera system has the sameapparatus configuration as the configuration illustrated in FIG. 1, adescription thereof will be omitted.

FIGS. 7A to 7C show flowcharts of an operation for obtaining aphotographed image according to the second embodiment. FIG. 7A shows aflowchart illustrating an operation of the entire camera system, FIG. 7Bshows a flowchart illustrating a pre-development operation, and FIG. 7Cshows a flowchart illustrating an operation of a defect correction unit.In FIGS. 7A to 7C, steps that perform the same operation as in the firstembodiment will be denoted with the same reference numerals as those ofFIGS. 3A to 3E.

As compared with the first embodiment that changes the interpolationmethod depending on whether the detected focus position is within arange of dmax, in the present embodiment, performs the image shift(pre-development) under the assumption that the focus position is withina range of dmax and determines the interpolation method in accordancewith the correlation of the shifted pixel value.

Steps of FIG. 7A will be described in order. Further, steps that are thesame as those in the flowchart of FIGS. 3A to 3E will be denoted withsame reference numerals, and a description thereof will be basicallyomitted.

Step S301 indicates starting of the image obtaining operation. In stepS302, after exposing the image pickup element 106 for an appropriatetime, an image pickup signal is read out and A/D converted to obtainimage data.

In step S703, a pre-development operation is performed, which isdifferent from the first embodiment. The pre-development will bedescribed below with reference to FIG. 7B.

In step S705, the defect correction unit is operated. The defectcorrection unit of this embodiment will be described below withreference to FIG. 7C.

Steps S306 to S309 perform the same operation as those in the firstembodiment. In other words, in step S306, the image shift unit isoperated, in step S307, the image generation unit is operated, in stepS308, the recording processing is performed and, then, in step S309, thesequence is completed.

The pre-development operation will be described with reference to FIG.7B.

In step S711, the pre-development operation starts and the sequenceproceeds to step S712.

Steps S712 to S714 are loop processings and images are generated(developed) in a plurality of set positions. As for the positions wherethe development is performed, a method of setting a plurality ofpositions within a range of |d_(max)| illustrated in the firstembodiment is considered to be used.

In step S713, the image shift unit operates so as to correspond to acurrent development position among the plurality of set positions. Thisoperation is the same operation described in the first embodiment.

Shift images are generated in the plurality of positions by theoperations performed in steps up to step S714. Since no image is addedby the image generation unit, pixels are shifted in the positions wherethe development is performed. Compared with FIGS. 5A to 5D, the imageshift on the re-formation plane of FIGS. 5A to 5D is completed, but aprocessing that obtains S_(i) by the addition in the vertical directionis not performed.

In step S715, the sequence returns to step S703 which is a caller of thepre-development routine.

The defect correction unit in the present embodiment will be describedwith reference to FIG. 7C. Step S371 indicates starting of the operationof the defect correction unit.

Steps S372 and S373 indicate the manipulation that reads out the defectinformation from the memory unit 108. As described in the firstembodiment, since the defect information is fixed, the defectinformation is stored to be used in this step.

Steps S374 to S378 are loop processings, and indicate that theinterpolation processing of the present embodiment is performed on allof the defective pixels.

In step S770, correlation of an angle (angle of incidence of the light)and a position (pixel value) is calculated. The correlation of the angleand the correlation of the position in this embodiment will be definedusing the reference numerals of FIGS. 5A to 5D.(angular correlation)_(p,i) =|X _(p−1,j) −X _(p+1,j)|(positional correlation)_(p,i) =|x _(p,i−1) −X _(p,i+1)|  (4)

Where, p and i are suffixes corresponding to the angle and the position,respectively. And, even though both the angle and the position areone-dimensionally illustrated for the convenience of the description,both may be considered as two-dimensional in the actual camera. In thiscase, it is possible to simply perform expansion. For example, if thesuffixes of the angle are p and q, and suffixes of the position are iand j, they may be represented by following equations.(angular correlation in p direction)_(p,q,i,j) =|X _(p−1,q,i,j) −X_(p+1,q,i,j)|(angular correlation in q direction)_(p,q,i,j) −|X _(p,q−1,i,j) −X_(p,q+1,i,j)|(positional correlation in i direction)_(p,q,i,j) =|X _(p,q,i−1,j) −X_(p,q,i+1,j)|(positional correlation in j direction)_(p,q,i,j) =|X _(p,q,i,j−1) −X_(p,q,i,j+1)|  (5)

In Equations 4 and 5, the correlation is calculated at adjacent angles(micro lenses), and in a positional (pixel) direction. If thecalculation method described above is used, there are advantages in thatthe correlation is simply calculated and the correlation of the positionand the correlation of the angle are treated in the same dimension.

Further, the arithmetic method of the correlation is not limitedthereto, but other methods may be used. For example, the correlation ofthe angle,(angular correlation)_(p,i)=Σ_(q)Σ_(r) |X _(q,1) −X _(r,i)|  (6)where summations are performed over q from 1 to N_(θ) and r from 1 toN_(θ), while excluding defective pixels.

According to Equation 6, it is possible to know how much the values ofthe pixels other than the defective pixels are similar to each other.Further, if Equation 6 is divided by an addition number to benormalized, the correlation of the angle may be treated in the samedimension as the correlation of the position.

In step S775, the correlation of the angle is compared with thecorrelation of the position. If the correlation value of the angle isequal to or less than the correlation value of the position, thesequence proceeds to step S778. In contrast, if the correlation of theangle is larger than the correlation of the position, the sequenceproceeds to step S776.

In step S776, the interpolation processing in the positional directionis performed, which is the same as the processing in step S376 of thefirst embodiment and performed by Equation 2.

In step S778, the interpolation processing in the angular direction isperformed, which may be defined by the following Equation.X _(p,i)=(X _(p−1,i) +X _(p+1,i))/2 (p−1≧ and p+1≦N _(θ))X _(p,i) =X _(p−1,i) (p+1>N)X _(p,i) =X _(p+1,i) (p−1<1)  (7)

If there is information on adjacent angles, information at both sides isused to perform the interpolation. If there is no information at bothsides, information at the other adjacent side is used to perform theinterpolation.

By the processings up to step S379, since the interpolation processingis performed using information having a good correlation (=highsimilarity), an appropriate interpolation processing is performed forthe defective pixel. As a result, it is possible to obtain a highquality image.

In step S380, the sequence returns to step S305, which is a caller ofthe defect correction routine.

Also, in the above-described present embodiment, when an image pickupelement that obtains information of the lights passing through differentpupil areas is used, even if the image pickup element includes the pixeldefect, it is possible to obtain a high quality image in which theinfluence of the defect is reduced.

As described above, even though the exemplary embodiments of the presentinvention have been described above, the present invention is notlimited to the exemplary embodiments, but may be modified or changedwithout departing from the gist of the invention.

Further, a program for achieving the functions of the processingsillustrated in FIGS. 3A to 3E and FIGS. 7A to 7C is recorded in acomputer readable recording medium, and the program recorded in therecording medium is read out and executed by a computer system toperform the processings. Here, the “computer system” includes anoperating system (OS) or hardware, such as peripheral equipment.Specifically, the program read out from the recording medium may bewritten in a memory provided in a functionality expansion board, whichis inserted in the computer or a functionality expansion unit connectedto the computer. A case when, on the basis of the instruction of theprogram, a CPU provided in the functionality expansion board or thefunctionality expansion unit performs a part of or all of the actualprocessings and, thus, the function of the above-described embodiment isachieved by the processings, is also included in the scope of thepresent invention.

Further, a “computer readable recording medium” refers to a portablemedium, such as a flexible disk, a magneto-optical disk, a ROM, or aCD-ROM, and a storage device, such as a hard disk, which is built in thecomputer system. Further, a computer readable recording medium mayinclude a device that stores a program for a predetermined time, such asa volatile memory (RAM) in a computer system that serves as a server ora client when a program is transmitted through a network, such as theInternet or a communication line, such as telephone line.

The program may be transmitted to other computer systems through atransmission medium from a computer system that stores the program in astorage device or by a transmitted wave in the transmission medium.Here, the “transmission medium” that transmits the program refers to amedium having a function that transmits information, such as a network(communication network), such as the Internet, or a communication line(communication wire), such as a telephone line.

Further, the program may achieve a part of the above-mentionedfunctions. Further, the program may be a differential file (differentialprogram) that implements the function by being combined with a programthat is already recorded in the computer system.

Further, a program product, such as the computer readable recordingmedium in which the program is recorded, may be applied as an exemplaryembodiment of the present invention. The program, the recording medium,the transmission medium, and the program product are also included inthe scope of the invention.

The present invention has been described with reference to exemplaryembodiments. The present invention is not limited to the above-describedembodiments, but various modifications are possible within the scopedescribed in the claims.

I claim:
 1. An image processing apparatus comprising: at least oneprocessor and a memory cooperatively coupled to function as: anobtaining unit that obtains an image signal of an image pickup elementhaving a pupil divider that restricts light of an optical image of anobject arriving at each pixel of the image pickup element to light froma specific pupil area of a photographing lens that photographs theobject; a memory unit that stores information of a pixel defect of theimage pickup element; an image shift unit that determines a shift amountof the image signal of the photographed object corresponding to an imagere-formation plane for every pupil area, to shift the image signal onthe basis of the image re-formation plane; a defect correction unit thatcorrects an image signal of a defective pixel using the shifted imagesignal of a pixel other than the defective pixel obtained by the imageshift unit, in accordance with the information of the pixel defect,wherein the defect correction unit calculates a correlation of aposition and an angle using the image signal obtained from the imageshift unit, and corrects the image signal of the pixel defect using theimage signal obtained from the image shift unit on the basis of thecalculated correlation; and an image re-formation unit that re-forms animage corresponding to the image re-formation plane from the imagesignal that is corrected by the defect correction unit.
 2. The imageprocessing apparatus according to claim 1, further comprising the imagepickup element that photoelectrically converts the optical image of theobject to be photographed, arriving through the photographing lens, andoutputs the image signal of the photographed object.
 3. The imageprocessing apparatus according to claim 1, further comprising a settingunit that sets the image re-formation plane on which an image isre-formed from the image signal of the photographed object, wherein theat least one processor functions as the setting unit.
 4. The imageprocessing apparatus according to claim 1, wherein the defect correctionunit corrects the image signal of the defective pixel by interpolationprocessing in an angular direction when a calculated correlation valueof the angle is equal to or less than a calculated correlation value ofthe position, and corrects the image signal of the defective pixel bythe interpolation processing in a positional direction when a calculatedcorrelation value of the angle is greater than a calculated correlationvalue of the position.
 5. The image processing apparatus according toclaim 1, further comprising a correlation calculation unit thatcalculates a difference of an image signal corresponding to a differentpupil area, using the image signal that is not shifted by the imageshift unit, wherein the image re-formation unit sets an imagere-formation plane based on the result of the correlation calculation ofthe correlation calculation unit, the at least one processor functionsas the correlation calculation unit.
 6. The image processing apparatusaccording to claim 1, wherein the memory unit stores a threshold valuedefined by the following equation:threshold value=(pupil division number)×(pixel interval of image pickupelement)/tan(angle difference from adjacent pupil area), and the imageprocessing apparatus further comprises a setting unit that includes aunit that sets the image re-formation plane within a range of thethreshold value, wherein the at least one processor functions as theunit of the setting unit.
 7. The image processing apparatus according toclaim 1, wherein the pupil divider is a micro lens array that isdisposed on a light receiving surface of the image pickup element, themicro lens array dividing a plurality of pixels formed on the lightreceiving surface of the image pickup element into a plurality of pixelgroups corresponding to each micro lens, and the micro lens allowingeach pixel of the corresponding pixel groups to correspond to the lightsfrom different pupil areas of the photographing lens.
 8. The imageprocessing apparatus according to claim 1, wherein the memory unitstores the threshold value defined by the following equation:threshold value=(pupil division number)×(pixel interval of image pickupelement)/tan(angle difference from adjacent pupil area), and the defectcorrection unit compares the image re-formation plane with the thresholdvalue and changes a method of correcting the image signal in accordancewith a comparison result.
 9. An image processing apparatus comprising:at least one processor and a memory cooperatively coupled to functionas: an obtaining unit that obtains an image signal of an image pickupelement having a pupil divider that restricts light of an optical imageof an object arriving at each pixel of the image pickup element to lightfrom a specific pupil area of a photographing lens that photographs theobject; a memory unit that stores information of a pixel defect of theimage pickup element; an image shift unit that determines a shift amountof the image signal of the photographed object corresponding to an imagere-formation plane for every pupil area, to shift the image signal onthe basis of the image re-formation plane; a defect correction unit thatcorrects an image signal of a defective pixel using the shifted imagesignal of a pixel other than the defective pixel obtained by the imageshift unit, in accordance with the information of the pixel defect; andan image re-formation unit that re-forms an image corresponding to theimage re-formation plane from the image signal that is corrected by thedefect correction unit, wherein the defect correction unit corrects theimage signal of the defective pixel using an output of the image shiftunit if the image re-formation plane is equal to or less than athreshold value, and corrects the image signal of the defective pixelusing an image signal other than the output of the image shift unit ifthe image re-formation plane is greater than the threshold value.
 10. Amethod of controlling an image processing apparatus, the methodcomprising: an obtaining step that obtains an image signal of an imagepickup element having a pupil divider that restricts light of an opticalimage of an object arriving at each pixel of the image pickup element tolight from a specific pupil area of a photographing lens thatphotographs the object; a storing step that stores information of apixel defect of the image pickup element; an image shift step thatdetermines a shift amount of the image signal of the photographed objectcorresponding to an image re-formation plane for every pupil area, toshift the image signal on the basis of the image re-formation plane; adefect correcting step that corrects an image signal of a defectivepixel using the shifted image signal of a pixel other than the defectivepixel, obtained in the image shift step, in accordance with theinformation of the pixel defect, wherein the defect correcting stepcalculates a correlation of the position and the angle using the imagesignal obtained from the image shift unit, and corrects the image signalof the pixel defect using the image signal obtained from the image shiftunit on the basis of the calculated correlation; and an imagere-formation step that re-forms an image corresponding to the imagere-formation plane from the image signal that is corrected in the defectcorrecting step.
 11. The method of controlling an image processingapparatus according to claim 10, further comprising photoelectricallyconverting, by the image pickup element, the optical image of the objectto be photographed, arriving through the photographing lens, andoutputting the image signal of the photographed object.
 12. The methodof controlling an image processing apparatus according to claim 10,further comprising a setting step that sets the image re-formation planeon which an image is re-formed from the image signal of the photographedobject.
 13. The method of controlling an image processing apparatusaccording to claim 10, wherein the defect correcting step corrects theimage signal of the defective pixel by interpolation processing in anangular direction when a calculated correlation value of the angle isequal to or less than a calculated correlation value of the position,and corrects the image signal of the defective pixel by theinterpolation processing in a positional direction when a calculatedcorrelation value of the angle is greater than a calculated correlationvalue of the position.
 14. The method of controlling an image processingapparatus according to claim 10, further comprising a correlationcalculation step that calculates a difference of an image signalcorresponding to a different pupil area, using the image signal that isnot shifted by the image shift unit, wherein the image re-formation stepsets an image re-formation plane based on the result of the correlationcalculation of the correlation calculation step.
 15. The method ofcontrolling an image processing apparatus according to claim 10, whereinthe storing step stores a threshold value defined by the followingequation:threshold value=(pupil division number)×(pixel interval of image pickupelement)/tan(angle difference from adjacent pupil area), the methodfurther comprising a setting step that sets the image re-formation planewithin a range of the threshold value.
 16. The method of controlling animage processing apparatus according to claim 10, wherein the pupildivider is a micro lens array that is disposed on a light receivingsurface of the image pickup element, the micro lens array dividing aplurality of pixels formed on the light receiving surface of the imagepickup element into a plurality of pixel groups corresponding to eachmicro lens, and the micro lens allowing each pixel of the correspondingpixel groups to correspond to the lights from different pupil areas ofthe photographing lens.
 17. The method of controlling an imageprocessing apparatus according to claim 10, wherein the storing stepstores the threshold value defined by the following equation:threshold value=(pupil division number)×(pixel interval of image pickupelement)/tan(angle difference from adjacent pupil area), and the defectcorrecting step compares the image re-formation plane with the thresholdvalue and changes a method of correcting the image signal in accordancewith a comparison result.
 18. A method of controlling an imageprocessing apparatus, the method comprising: an obtaining step thatobtains an image signal of an image pickup element having a pupildivider that restricts light of an optical image of an object arrivingat each pixel of the image pickup element to light from a specific pupilarea of a photographing lens that photographs the object; a storing stepthat stores information of a pixel defect of the image pickup element;an image shift step that determines a shift amount of the image signalof the photographed object corresponding to an image re-formation planefor every pupil area, to shift the image signal on the basis of theimage re-formation plane; a defect correction step that corrects animage signal of a defective pixel using the shifted image signal of apixel other than the defective pixel obtained by the image shift unit,in accordance with the information of the pixel defect; and an imagere-formation step that re-forms an image corresponding to the imagere-formation plane from the image signal that is corrected by the defectcorrection unit, wherein the defect correcting step corrects the imagesignal of the defective pixel using an output of the image shift unit ifthe image re-formation plane is equal to or less than a threshold value,and corrects the image signal of the defective pixel using an imagesignal other than the output of the image shift unit if the imagere-formation plane is greater than the threshold value.
 19. Anon-transitory computer readable storing medium storing a programcomprising a code for causing a computer to execute a method forcontrolling an image processing apparatus, the method comprising: anobtaining step that obtains an image signal of an image pickup elementhaving a pupil divider that restricts light of an optical image of anobject arriving at each pixel of the image pickup element to light froma specific pupil area of a photographing lens that photographs theobject; a storing step that stores information of a pixel defect of theimage pickup element; an image shift step that determines a shift amountof the image signal of the photographed object corresponding to an imagere-formation plane for every pupil area, to shift the image signal onthe basis of the image re-formation plane; a defect correcting step thatcorrects an image signal of a defective pixel using the shifted imagesignal of a pixel other than the defective pixel, obtained in the imageshift step, in accordance with the information of the pixel defect,wherein the defect correcting step calculates a correlation of theposition and the angle using the image signal obtained from the imageshift unit, and corrects the image signal of the pixel defect using theimage signal obtained from the image shift unit on the basis of thecalculated correlation; and an image re-formation step that re-forms animage corresponding to the image re-formation plane from the imagesignal that is corrected in the defect correcting step.
 20. Thenon-transitory computer readable storing medium according to claim 19,the method further comprising photoelectrically converting, by the imagepickup element, the optical image of the object to be photographed,arriving through the photographing lens, and outputting the image signalof the photographed object.
 21. The non-transitory computer readablestoring medium according to claim 19, the method further comprising asetting step that sets the image re-formation plane on which an image isre-formed from the image signal of the photographed object.
 22. Thenon-transitory computer readable storing medium according to claim 19,wherein the defect correcting step corrects the image signal of thedefective pixel by interpolation processing in an angular direction whena calculated correlation value of the angle is equal to or less than acalculated correlation value of the position, and corrects the imagesignal of the defective pixel by the interpolation processing in apositional direction when a calculated correlation value of the angle isgreater than a calculated correlation value of the position.
 23. Thenon-transitory computer readable storing medium according to claim 19,the method further comprising a correlation calculation step thatcalculates a difference of an image signal corresponding to a differentpupil area, using the image signal that is not shifted by the imageshift unit, wherein the image re-formation step sets an imagere-formation plane based on the result of the correlation calculation ofthe correlation calculation step.
 24. The non-transitory computerreadable storing medium according to claim 19, wherein the storing stepstores a threshold value defined by the following equation:threshold value=(pupil division number)×(pixel interval of image pickupelement)/tan(angle difference from adjacent pupil area), the methodfurther comprising a setting step that sets the image re-formation planewithin a range of the threshold value.
 25. The non-transitory computerreadable storing medium according to claim 19, wherein the pupil divideris a micro lens array that is disposed on a light receiving surface ofthe image pickup element, the micro lens array dividing a plurality ofpixels formed on the light receiving surface of the image pickup elementinto a plurality of pixel groups corresponding to each micro lens, andthe micro lens allowing each pixel of the corresponding pixel groups tocorrespond to the lights from different pupil areas of the photographinglens.
 26. The non-transitory computer readable storing medium accordingto claim 19, wherein the storing step stores the threshold value definedby the following equation:threshold value=(pupil division number)×(pixel interval of image pickupelement)/tan(angle difference from adjacent pupil area), and the defectcorrecting step compares the image re-formation plane with the thresholdvalue and changes a method of correcting the image signal in accordancewith a comparison result.
 27. A non-transitory computer readable storingmedium storing a program comprising a code for causing a computer toexecute a method for controlling an image processing apparatus, themethod comprising: an obtaining step that obtains an image signal of animage pickup element having a pupil divider that restricts light of anoptical image of an object arriving at each pixel of the image pickupelement to light from a specific pupil area of a photographing lens thatphotographs the object; a storing step that stores information of apixel defect of the image pickup element; an image shift step thatdetermines a shift amount of the image signal of the photographed objectcorresponding to an image re-formation plane for every pupil area, toshift the image signal on the basis of the image re-formation plane; adefect correcting step that corrects an image signal of a defectivepixel using the shifted image signal of a pixel other than the defectivepixel, obtained in the image shift step, in accordance with theinformation of the pixel defect; and an image re-formation step thatre-forms an image corresponding to the image re-formation plane from theimage signal that is corrected in the defect correcting step, whereinthe defect correcting step corrects the image signal of the defectivepixel using an output of the image shift unit if the image re-formationplane is equal to or less than a threshold value, and corrects the imagesignal of the defective pixel using an image signal other than theoutput of the image shift unit if the image re-formation plane isgreater than the threshold value.